1. Field of the Invention
The present invention relates to calibrating placement information of markers, provided on an imaging device, as to the imaging device.
2. Description of the Related Art
In recent years, studies regarding Mixed Reality, which aims to seamlessly join real space and virtual space, have been actively pursued. An image display device for presenting this Mixed Reality is realized by a video see-through method for displaying a synthesized image obtained by superimposing and drawing an image in virtual space (a virtual object, character information, or the like, drawn with computer graphics) generated according to the position and orientation of a later-described imaging device over an image in real space photographed using an imaging device such as a video camera.
There are expectations for application of such image display devices to various fields unlike those for conventional virtual reality, such as surgical aids for superimposing and displaying the situation within a patient's body over the patient's body surface, Mixed Reality games where a player would battle with a virtual enemy in real space, and so forth.
The most important problem to be solved for all such applications is how to accurately perform the registration between real space and virtual space, and heretofore, many measures have been attempted. The registration problem in Mixed Reality is concluded in the problem of obtaining the position and orientation of an imaging device in a scene (i.e., in the world coordinates system).
A commonly-employed method for solving this problem is to dispose or set multiple markers in the scene, and use the coordinates of the markers in the world coordinates system, and the coordinates of the projected images of the markers within an image photographed by an imaging device, so as to obtain the position and orientation of the imaging device in the scene. A method for calculating the position and orientation of an imaging device based on a pair of the coordinates in the world coordinates system of a marker in a scene and image coordinates of a projected image thereof has been proposed in the field of photographic measurement as of old.
Also, technology has been realized in which the position and orientation of an object is obtained by setting multiple markers with regard to an object to be measured, photographing the object with an externally disposed bird's-eye view camera, and detecting the image coordinates of the projected images of the markers within the taken bird's-eye view image (see R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, Vol. 13, No. 3, pp. 331-356, 1994; and D. G. Lowe: “Fitting parameterized three-dimensional models to images”, IEEE translations on PAMI, Vol. 13, No. 5, pp. 441-450, 1991).
Further, the present inventors have also proposed in US Published Patent Application No. 20040176925 (filed in the USPTO on Jan. 8, 2004) a measurement method for the position and orientation of an imaging device, which integrates: a method for calculating the position and orientation of an imaging device by detecting projected images of markers within a scene, from an image which the imaging device, to be measured itself, has taken; and a method for calculating the position and orientation of an imaging device by detecting projected images of markers set upon the imaging device itself, from a bird's-eye view image taken of the object to be measured from a bird's-eye view position.
However, with the method for calculating the position and orientation of an imaging device by detecting image coordinates of projected images of markers within a bird's-eye view image, the relative positional relation between the multiple makers set upon the object to be measured, and the object to be measured, must be known. In the event that the relative positional relation of each of the markers is unknown, e.g., in the event that markers of which the features can be described as a single point are each placed upon the object to be measured, such as with a case of utilizing the center-of-gravity position of projected images of a markers as features using colored spherical markers or circular markers, the three-dimensional position of each of the markers in the coordinate system of the object to be measured must be measured beforehand. On the other hand, in the event that the relative positional relation between the markers is known, such as with a case of the markers (even if markers such as described above are used) having been mounted to a jig serving as a base beforehand, the positions of each of the markers on the coordinate system of this jig (hereafter, a coordinates system for describing the relative position of markers in this way is referred to as “marker coordinate system”) having been measured beforehand, and the jig having been mounted to the object to be measured, all that is necessary is for the position and the orientation for this jig in the coordinate system of the object to be measured to be known. It should be noted that in the event that the features of the markers are described with multiple points, e.g., in the event of using the vertices of the projected images of markers having known shapes such as squares or triangles, for example, as features, each marker is interpreted as being a group of multiple markers, and is taken as being a case wherein “the relative positional relation between markers is known”. However, no method for accurately calibrating these is commonly known, and conventionally, there has been no other way but to use the inaccurate position or position and orientation obtained by manual measurement as known values. Accordingly, there is room for improvement in the above-described method for detecting the position and orientation of an object by measuring image coordinates of projected images of markers within a bird's-eye view image, in that the position and orientation of the object could only be measured with low precision from a practical standpoint.